Causal inference using observational intensive care unit data: a scoping review and recommendations for future practice

This scoping review focuses on the essential role of models for causal inference in shaping actionable artificial intelligence (AI) designed to aid clinicians in decision-making. The objective was to identify and evaluate the reporting quality of studies introducing models for causal inference in intensive care units (ICUs), and to provide recommendations to improve the future landscape of research practices in this domain. To achieve this, we searched various databases including Embase, MEDLINE ALL, Web of Science Core Collection, Google Scholar, medRxiv, bioRxiv, arXiv, and the ACM Digital Library. Studies involving models for causal inference addressing time-varying treatments in the adult ICU were reviewed. Data extraction encompassed the study settings and methodologies applied. Furthermore, we assessed reporting quality of target trial components (i.e., eligibility criteria, treatment strategies, follow-up period, outcome, and analysis plan) and main causal assumptions (i.e., conditional exchangeability, positivity, and consistency). Among the 2184 titles screened, 79 studies met the inclusion criteria. The methodologies used were G methods (61%) and reinforcement learning methods (39%). Studies considered both static (51%) and dynamic treatment regimes (49%). Only 30 (38%) of the studies reported all five target trial components, and only seven (9%) studies mentioned all three causal assumptions. To achieve actionable AI in the ICU, we advocate careful consideration of the causal question of interest, describing this research question as a target trial emulation, usage of appropriate causal inference methods, and acknowledgement (and examination of potential violations of) the causal assumptions.


Reference
Supplementary Table 4: Subcomponent-specific results of the quality of reporting assessment in the reproducibility domain, specifically for the studies using the parametric G formula (n=5).

Treatment strategies
Outcome Follow-up period Analysis plan (parametric G formula)

Method to evaluate the G formula
Agodi 2017 Supplementary Table 5: Subcomponent-specific results of the quality of reporting assessment in the reproducibility domain, specifically for the studies using reinforcement learning (n=31 Describe the rationale for the review in the context of what is already known.Explain why the review questions/objectives lend themselves to a scoping review approach.

8-10
Objectives 4 Provide an explicit statement of the questions and objectives being addressed with reference to their key elements (e.g., population or participants, concepts, and context) or other relevant key elements used to conceptualize the review questions and/or objectives.

Protocol and registration
5 Indicate whether a review protocol exists; state if and where it can be accessed (e.g., a Web address); and if available, provide registration information, including the registration number.

22
Eligibility criteria 6 Specify characteristics of the sources of evidence used as eligibility criteria (e.g., years considered, language, and publication status), and provide a rationale.

Information sources*
7 Describe all information sources in the search (e.g., databases with dates of coverage and contact with authors to identify additional sources), as well as the date the most recent search was executed.
22 Search 8 Present the full electronic search strategy for at least 1 database, including any limits used, such that it could be repeated.

Supplementary Table 8
Selection of sources of evidence †

9
State the process for selecting sources of evidence (i.e., screening and eligibility) included in the scoping review.

Data charting process ‡ 10
Describe the methods of charting data from the included sources of evidence (e.g., calibrated forms or forms that have been tested by the team before their use, and whether data charting was done independently or in duplicate) and any processes for obtaining and confirming data from investigators.

Data items 11
List and define all variables for which data were sought and any assumptions and simplifications made.

Critical appraisal of individual 12
If done, provide a rationale for conducting a critical appraisal of included sources of evidence;

24-25 sources of evidence §
describe the methods used and how this information was used in any data synthesis (if appropriate).

13
Describe the methods of handling and summarizing the data that were charted.

Selection of sources of evidence 14
Give numbers of sources of evidence screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally using a flow diagram.

Characteristics of sources of evidence 15
For each source of evidence, present characteristics for which data were charted and provide the citations.If done, present data on critical appraisal of included sources of evidence (see item 12).

Results of individual sources of evidence 17
For each included source of evidence, present the relevant data that were charted that relate to the review questions and objectives.

18
Summarize and/or present the charting results as they relate to the review questions and objectives.

Summary of evidence 19
Summarize the main results (including an overview of concepts, themes, and types of evidence available), link to the review questions and objectives, and consider the relevance to key groups.

13-15
Limitations 20 Discuss the limitations of the scoping review process. 21

21
Provide a general interpretation of the results with respect to the review questions and objectives, as well as potential implications and/or next steps.* Where sources of evidence (see second footnote) are compiled from, such as bibliographic databases, social media platforms, and Web sites.† A more inclusive/heterogeneous term used to account for the different types of evidence or data sources (e.g., quantitative and/or qualitative research, expert opinion, and policy documents) that may be eligible in a scoping review as opposed to only studies.This is not to be confused with information sources (see first footnote).‡ The frameworks by Arksey and O'Malley (6) and Levac and colleagues (7) and the JBI guidance (4, 5) refer to the process of data extraction in a scoping review as data charting.§ The process of systematically examining research evidence to assess its validity, results, and relevance before using it to inform a decision.This term is used for items 12 and 19 instead of "risk of bias" (which is more applicable to systematic reviews of interventions) to include and acknowledge the various sources of evidence that may be used in a scoping review (e.g., quantitative and/or qualitative research, expert opinion, and policy document). From

Subcomponent
Leading question Eligibility criteria -Are eligibility criteria for target population described?
Treatment strategies -Are the compared regimes described in such a way that one can think of an analogue randomized trial (ie, target trial)?
Outcome -Is the considered patient outcome described?Follow-up period time-zero Is the time-zero (baseline) explicitly mentioned or can it reasonably be assumed from the data collection description?
Follow-up Are the start and end of follow-up period explicitly mentioned or can these reasonably be assumed from the data collection description?
Time-resolution Is the size of the considered time steps (ie, the timeresolution) explicitly mentioned or can it reasonably be assumed from the data collection description?Analysis plan (RL) Learning scheme Is the learning scheme used to train the RL agent described?
State space model Does the methods description specify whether continuous or categorical state space and on which variables states were based?Environment model Is the modelling of environment described (or clearly not applicable, eg, with model-free learning schemes)?
Discount factor Is the used discount factor described?

Analysis plan (parametric G formula)
Outcome estimator Is the model used to estimate the outcome described?(eg, logistic regression) Outcome predictors Are variables/features used to model the outcome described (including both time-fixed and time-varying variables)?

Confounders estimators
Is the model used to estimate the confounders described?(eg, logistic regression) Confounders predictors Are variables/features used to model the confounders described?Method to evaluate the G formula Is the method to evaluate the G formula described?(eg, Monte-Carlo sampling)

Propensity score estimator
Is the model used to estimate the propensity score described?(eg, logistic regression) Propensity score predictors Are variables/features used to model the propensity score described (including both time-fixed and time-varying variables)?

22 FUNDING Funding 22
Describe sources of funding for the included sources of evidence, as well as sources of funding for the scoping review.Describe the role of the funders of the scoping review.26 JBI = Joanna Briggs Institute; PRISMA-ScR = Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews.

Table 3 :
Subcomponent-specific results of the quality of reporting assessment in the reproducibility domain, specifically for the studies inverse-probability-of-treatment weighting or targeted minimum loss-based estimation (n=43).

Table 6 :
reporting of assumptions assessment results per study.IPT=inverse probability of treatment